With rapid development of the Internet and Web 2.0, web (Web) applications have created tens of billions of small files. Moreover, people are uploading more pictures, videos, and music, and are sending hundreds of billions of emails, which generates more data. According to statistics of the Internet Data Center (IDC), a data volume is expected to grow by 44 times in the next 10 years, and global data will grow to 35 ZB (Zettabyte) by 2020, where 80% is unstructured data, and a large portion is inactive data. There is a problem about how to store such mass data.
For such a huge data volume, a block storage technology and a file storage technology that only have scalability in PB (Petabyte) seem powerless. An object-based storage system (Object-Based Storage System), with advantages such as direct access of the block storage technology and data sharing of the file storage technology, provides a highly-reliable, cross-platform, and secure data-sharing storage system.
As shown in FIG. 1, an object-based storage system generally includes a controller 101, a storage device client 103, and a storage device cluster 104. The controller 101 may be further connected to an external client, and the external client may be connected to another device (for example, a domain name server) (the external client and the domain name server are not shown in FIG. 1). The storage device client 103 is connected to multiple storage devices in the storage device cluster 104, and a storage device may be an IP (internet protocol) hard disk or another smart disk, for example, a solid state drive (Solid State Drive, SSD).
In an object-based storage system, a two-layer service model based on a container (bucket) and an object (object) is used most widely. The container may be understood as a special top-level directory or a globally unique domain name. The object is a basic unit for object storage. Each object is a combination of data and a data attribute set. A data attribute may be set as required of an application, and includes data distribution, quality of service, and the like. The object maintains its own attribute, which simplifies a management task of a storage system and improves flexibility. The object may have a different size, and may include an entire data structure, for example, a file, a database entry, a directory, and the like.
If a container operation (which may also be referred to as a bucket operation) is to be performed, a user sends a container IO (input/output) request to a controller by using a client; when an object operation is to be performed, the client sends an object IO request to the controller. Compared with the object IO request, the container IO request is generally smaller, and generally does not exceed 1 megabyte (M). The object IO request is generally larger, and a large one may be several M or may be less than 1 M. In embodiments of the present invention, processing may be performed only according to a size of an IO request regardless of whether the IO request is a container IO request or an object IO request. Therefore, in the following descriptions, an IO request is not specifically referred to as a container IO request or an object IO request, but is simply referred to as an object IO request.
An object IO request may be a write object IO request or a read object IO request. In the embodiments, unless otherwise specified, a described procedure is applicable to both a write object IO request and a read object IO request.
When the user needs to read and write data, the external client is used to send an object IO request to the controller 101; then, the controller 101 sends the object IO request to the storage device client 103; the storage device client 103 sends the object IO request to a corresponding storage device by using an algorithm; and the storage device processes the object IO request. Each storage device can process and temporarily store a limited quantity of object IO requests. During peak service hours, the following case is likely to occur: object IO requests are delivered at a speed exceeding a processing capability and a cache capability of each storage device. For an object IO request that cannot be processed, the storage device further needs to feed back an overload response. A resource of the storage device that is occupied for feeding back the overload response is not less than a resource required to process one object IO request. This may result in that more object IO requests cannot be processed.
In addition, a large object IO request is generally split into multiple small object IO requests; the large object IO request fails to be processed provided that one small object IO request fails to be processed. A small object IO request usually needs to wait for a long time, and a probability of a processing failure increases after waiting for a long time.
An object IO request that fails to be processed and an object IO request that cannot be processed in a timely manner will be resent to the storage device in repeated attempts for processing, which forms a vicious cycle and reduces performance and a success rate of the entire object-based storage system for processing an object IO request.